Touchless interfaces

ABSTRACT

The shape or position of an object is estimated using a device comprising one or more transmitters and one or more receivers, forming a set of at least two transmitter-receiver combinations. Signals are transmitted from the transmitters, through air, to the object. They are reflected by the object and received by the receivers. A subset of the transmitter-receiver combinations which give rise to a received signal meeting a predetermined clarity criterion is determined. The positions of points on the object are estimated using substantially only signals from the subset of combinations.

RELATED APPLICATIONS

This application is a Continuation of application Ser. No. 13/503,347,filed Aug. 14, 2012, which is the National Stage of InternationalApplication No. PCT/GB2010/051789, filed Oct. 25, 2010, which claims thebenefit of GB 0918596.8, filed Oct. 23, 2009 and GB 1001733.3, filedFeb. 3, 2010. Each of these applications is hereby expresslyincorporated by reference in its entirety herein.

This invention relates to touchless interfaces employing transducers todetermine information about an object from reflections from the objectof signals, particularly although not exclusively, ultrasonic signals.

It is known for an electronic device, such as a desktop computer or amobile telephone, to determine the motion of an object, such as a humanhand, by transmitting signals from one or more transmitters andreceiving reflections of the signals from the object at one or morereceivers, for the purpose of controlling the device. For example,WO2009/115799 describes some such devices.

Given that at least some previous proposals require the inclusion ofadditional hardware in the form of ultrasonic transducers, it is naturalto make full use of them, particularly where the application calls foran image of the object to be built up (as opposed say to recognition ofa simple motion tendency). Where a device has more than one transmitterand/or more than one receiver, the presumption has typically been toutilise all available transmitter-receiver pairs (or channels). In someapplications (e.g. medical imaging) this can be done by combining thesignals received by all available receivers (e.g. by averaging orsumming the signals received by each). However, such processing can beinefficient and computationally costly.

When viewed from a first aspect, the present invention provides a methodof estimating the shape and/or position of an object using a devicecomprising one or more transmitters and one or more receivers, forming aset of at least two transmitter-receiver combinations, said methodcomprising:

transmitting signals from the transmitter(s), said signals travellingthrough air to the object, being reflected by the object and received bythe receiver(s);

determining a subset of the transmitter-receiver combinations which giverise to a received signal meeting a predetermined clarity criterion; and

estimating positions for a plurality of points on said object usingsubstantially only signals from said subset of combinations.

The invention extends to a device configured to estimate the shapeand/or position of an object, the device comprising:

one or more transmitters and one or more receivers forming a set of atleast two transmitter-receiver combinations, said transmitters beingarranged to transmit signals;

means for determining a subset of the transmitter-receiver combinationswhich give rise to a received signal meeting a predetermined claritycriterion; and

means for estimating positions for a plurality of points on said objectusing substantially only signals from said subset of combinations.

The invention also extends to a computer software product, and to acarrier bearing the same, configured, when run on a device comprisingone or more transmitters and one or more receivers, forming a set of atleast two transmitter-receiver combinations, to operate the device toestimate the shape and/or position of an object, the software productcomprising:

instructions for transmitting signals from the transmitter(s);

logic for determining a subset of the transmitter-receiver combinationswhich give rise to a received signal meeting a predetermined claritycriterion; and

logic for estimating positions for a plurality of points on said objectusing substantially only signals from said subset of combinations.

The skilled person will appreciate that, by using only signals fromtransmitter-receiver pairs that have a clear ‘view’ of the object, andby determining the position of more than one point on the object, theaccuracy of the resulting position or shape estimates can besubstantially improved. In particular, signals from transmitter-receivercombinations which do not meet the clarity criterion are excluded,resulting in enhanced performance. This can be contrasted with anapproach in which signals from all transmitter-receiver pairs are usedto determine a position estimate of a single point on an object, forexample by averaging information relating to signals received on allavailable channels.

Moreover, the number of transmitter-receiver combinations that areinvolved in the estimation is less than if all availabletransmitter-receiver pairs are used, and thus the processingrequirements can be significantly lower.

The object is not limited to any particular class or type of object, butin some embodiments it is a pointing or gesticulating implement such asa stylus, a human hand, or an extended human finger. The object may bejust a part of a larger object, such as the tip of a finger. In someembodiments, the shape and/or position of multiple objects is estimatedsubstantially simultaneously.

In one set of embodiments the clarity criterion is defined such that itis met by a transmitter-receiver combination when no other reflectivesurface is at the same or similar time-of-flight distance as the objectfor that combination; conversely, the criterion is preferably such thatit is not met when another reflective surface is at the same or similartime-of-flight distance from the combination as the object. Twotime-of-flight distances may be similar if their difference is less apredetermined amount, e.g. less than 5 centimeters, or less than 1centimeter.

WO 2009/115799, which is hereby incorporated by reference in itsentirety, describes various approaches for analysing received signals,including calculating impulse response images and applying filters tothese. This is described in more detail below.

In some embodiments, the clarity criterion is implemented by determiningwhether the received signals for a particular transmitter-receivercombination have a clear leading edge; this may be implemented bydetecting a match with a leading-edge filter. Such a filter wouldtypically require a certain level of contrast over a certain time toindicate the initial absence of signal followed by presence of asuitable signal. Clearly the particular details of such a filter can bechosen, without the exercise of inventive skill, to suit the particularapplication.

The transmit signals could be optical or other electromagnetic signals,but they are preferably acoustic, more preferably ultrasonic havingfrequencies greater than 20 kHz, especially greater than 30 kHz. In someembodiments the frequency might be in the range 35-45 kHz. In otherembodiments a higher frequency than this could be used. Thus in anotherset of embodiments the frequency is greater than 50 kHz or even greaterthan 100 kHz—e.g. between 100 and 200 kHz. The transmitters could becontrolled to transmit continuous signals or discrete impulses. Thesignal may comprise a single frequency, or may comprise a plurality offrequencies.

The nature of the transmit signals can be selected as appropriate. In asimple embodiment they could comprise a single impulse or spike, i.e.approximating a Dirac delta function within the limitations of theavailable bandwidth. This has some advantages in terms of requiringlittle, if any, processing of the ‘raw signal’ to calculate impulseresponses (in the theoretical case of a pure impulse, no calculation isrequired) but gives a poor signal-to-noise ratio because of thedeliberately short transmission. In other embodiments the transmitsignals could be composed of a series or train of pulses. This gives abetter signal-to-noise ratio than a single pulse without greatlyincreasing the computation required. Pulse trains can be used to computeimpulse responses at comparatively low processing cost because manyelements in the correlation chain are zeros, hence the correspondingsteps in the multiply-and-sum operations, i.e. the convolution, can beskipped. Often, this can reduce the overall processing cost by 80% or90%. In other embodiments the transmit signals comprise one or morechirps—i.e. a signal with rising or falling frequency. These give a goodsignal-to-noise ratio and are reasonable for calculating the impulseresponses using a corresponding de-chirp function applied to the ‘raw’received signal.

The recited transmitter-receiver combinations may comprise a singletransmitter and a single receiver. In a set of preferred embodimentshowever transmitter-receiver combinations comprising a plurality oftransmitters or receivers are used. In one such set of embodimentscombinations including a plurality of receivers are used which allowsdirectional information for the reflected signals to be derived bycomparing the small time delays in receipt of the signal at therespective receivers (coupled of course with knowledge of the respectivepositions of the receivers on the device).

In another (overlapping) set of embodiments the transmitter-receivercombinations used comprise a plurality of transmitters (at least some ofwhich could be the same physical transducers as the receivers referredto above, in embodiments employing transducers which can be used astransmitters or receivers). Such plural transmitters can be usedtogether to transmit the same signal with relative time delays in orderto ‘steer’ the transmission in a certain direction. This could be usefulwhere an object has already been detected at an approximate location orin an approximate direction to direct energy in that direction and thusobtain more detailed or accurate information. It also has the beneficialeffect of reducing the probability of unwanted reflections from otherobjects interfering with the signals of interest, i.e. it improves thesignal-to-noise ratio.

In either case above the greater the number of transducers, the moreeffective this approach can be. Also, having the transducers closetogether makes this approach more effective. Preferably therefore atleast some of the transmitters or receivers are provided in a group inwhich the maximum distance between any transmitter or receiver and itsnearest neighbour is less than the longest wavelength transmitted byeither of them, or by the device, in normal use; more preferably, lessthan half such a wavelength. Similarly it is preferred that at leastsome of the transmitters or receivers are provided in a group in whichthe maximum distance between any transmitter or receiver and its nearestneighbour is less than the longest wavelength transmitted by either ofthem, or by the device, in normal use; more preferably, less than halfsuch a wavelength. This allows the respective group to act as aphased-array and to direct or receive energy preferentially from aparticular direction compared with other directions. Such a group can,in some embodiments, be considered as a single directional transducer.In some situations it may be desirable that the maximum distance betweentwo transmitters or receivers be less than the smallest wavelengthtransmitted by either of them, or by the device, in normal use;sometimes less than half the smallest wavelength.

In one set of preferred embodiments the device is arranged to conduct apreliminary step in which a plurality, preferably all, of thetransmitters transmit respective signals in sequence. A plurality,preferably all, of the receivers may then be used to determinetransmitter-receiver pairs for which the clarity criterion is met—e.g.by having a sufficiently well-defined leading edge. These could be usedto construct a two-dimensional matrix with entries indicating ‘clear’pairings. The transmitter-receiver pairs may then be used for subsequentimaging. Of course rather than transmitter-receiver pairs, combinationscomprising a plurality of transmitters and/or a plurality of receiverscould be used for this step as described above, although ifsingle-transmitter-receiver pairs are used, combinations with multipletransmitters and/or receivers may still be used for subsequent imaging.

Whether or not groups of transmitters or receivers are used in concertwith each other, and whether or not such groups are closely spaced,their physical layout can be chosen to suit the application. However, ina set of preferred embodiments the transmitters and/or receivers arelaid out in a regularly-spaced, preferably rectangular array. In oneparticular set of preferred embodiments such an array comprises at leasttwo parallel rows of transducers. This has many potential benefits—e.g.in providing strong directivity in directions normal to the rows; thishas been found to give specific benefits in the context of a hand-heldmobile device since it provides a simple mechanism for strongly avoidinginterference arising as a result of reflections from the hand holdingthe device.

Having two or more rows also allows a local two-dimensional “cluster” ofelements to be defined. If such a cluster has a clear view of a part ofthe object, its transducers can be used to “image” the object ofinterest, taking advantage of the object being clearly visible, i.e. nothampered by overlapping signals from other objects. That is, if atwo-dimensional cluster of receivers can clearly see the object ofinterest, all the necessary parameters for estimating the object's anglerelative to this cluster could be determined. By contrast, if the devicecomprised a single ring of receivers, there would be an ambiguity inthese angle estimates, i.e. it would not be straightforward to estimateboth azimuth and elevation information.

More generally, however, a two-dimensional local cluster of transducerscould be defined of any suitable pattern (e.g. a square, a circle, anellipse, a sparse matrix or several sparse or dense matrices of anyshape, taken from the overall layout of transducers). A transducer couldsometimes be used as both a transmitter or a receiver, interchangeably.In some situations, i.e. when the object is close to a two-dimensionallocal cluster, three-dimensional position information for the object canbe obtained using only the transducers of the local cluster. When theobject is further away, the local cluster can provide accurate angularpositioning (azimuth and elevation). To obtain the range to the object,an additional transmit channel, either within the local cluster oroutside it, could be used to obtain an accurate 3D position for theobject, consisting of two angles and a range, or alternatively alocation in three-dimensional space. The roles of a local,two-dimensional receive cluster and a two-dimensional transmit clustermay also be reversed, by using scanning or transmit-beamformingtechniques.

Mobile devices with two parallel rows of acoustic transducer elementsare believed to be novel and inventive in their own right and thus, whenviewed from a further aspect, the invention provides a handheld devicecomprising a plurality of acoustic transducers as part of a userinterface, said transducers being arranged in at least two parallelrows.

From a further aspect, the invention provides a method of operating sucha handheld device to estimate the shape and/or position of an object,and/or to characterise the motion of an object. The invention alsoextends to a computer software product, and to a carrier bearing thesame, configured, when run on such handheld device, to cause it to carryout such a method.

In a preferred set of embodiments the rows are parallel with an edge ofthe device. Preferably the device comprises a display or interactionsurface and the rows are both on the same side of it—i.e. one of therows lies between the display or interaction surface and the other row.An interaction surface might be a touch-sensitive panel, such as atouchpad, or it could simply be an area designated for interaction.

Preferably each row comprises at least three such elements. Preferablyeach row is longer than the distance between the rows.

Conveniently the rows are substantially straight, but this is notessential, e.g. they may be smoothly curving. The transducers may belocated strictly along the row, but may also be staggered variously toone side or the other; for example, in a zigzag fashion. This has theadvantage of reducing the spacing between elements in some directions,further improving the beamforming aspects of the solution.

The display or interaction surface may be of any shape, but ispreferably a flat rectangle. In this case, the two rows preferably liein substantially the same plane as the rectangle (albeit that they maybe partly or completely recessed below the plane) and are preferablyparallel to an edge of the rectangle.

The transducers are preferably substantially regularly spaced along therows. The distance between adjacent transducers in some embodiments isequal to or less than half the maximum or average (mean, median or mode)or highest or base wavelength transmitted by one or more of thetransducers in use (i.e. less than λ/2). However, the spacing may alsobe much larger than this, which can be easier to implement and mightprovide either aesthetic or design benefits.

The distance between the rows is not limited to any particular size;however, in some preferred embodiments it is equal to or less than λ/2.

Although only two rows have so far been discussed, it will be readilyappreciated that more than two rows may be provided; for example, three,four, five or more.

In particularly preferred embodiments, the screen or interaction surfaceis surrounded by two concentric rings of transducer elements, preferablyconforming to the shape of the screen or surface. Where the screen orsurface is rectangular, it is preferably surrounded by two rectangularrings of transducers, one larger than the other. The rings may extendbeyond the screen at the corners, e.g. onto the sides of the device, ormay have gaps at the corners. Of course, in some embodiments, three,four, five or more concentric rings may be provided.

In some embodiments, the display or interaction surface is arranged totransmit an acoustic signal. Various arrangements are possible. In someembodiments the apparatus comprises a transmitting surface comprising adisplay screen or disposed in substantially overlapping relationshipwith a display screen. Other transducer elements may be provided asexplained above; for example, the transmitting surface may be surroundedby a plurality of concentric rings of acoustic receivers. The othertransducer elements may all be receivers, or may comprise bothtransmitters and receivers. Methods of the invention may comprisetransmitting an acoustic signal from a transmitting surface, thetransmitting surface comprising a display screen or being disposed insubstantially overlapping relationship with a display screen.

The transmitting surface may also be a receiving surface, arranged toreceive a reflection of an acoustic signal from an object of interest.

From a further aspect, the invention provides a method of characterisingthe shape, location or motion of an object, comprising:

transmitting an acoustic signal from a transmitting surface, thetransmitting surface comprising a display screen or being disposed insubstantially overlapping relationship with a display screen;

receiving a reflection of the signal from an object which is directly infront of the display screen; and

characterising the shape, location or motion of the object usinginformation determined from the received signal.

From another aspect, the invention provides apparatus for characterisingthe shape, location or motion of an object, comprising:

a display screen;

a transmitting surface comprising the display screen, or being disposedin substantially overlapping relationship with the display screen, andarranged to transmit an acoustic signal from substantially all of itssurface;

a receiver arranged to receive a reflection of the acoustic signal froman object which is directly in front of the display screen; and

means for characterising the shape, location or motion of the objectusing information determined from the received signal.

Such an arrangement is advantageous in that it can reduce or obviate theneed for separate transmitters located apart from the display screen.This can lead to a more attractive design than is otherwise possible;especially when the display screen spans substantially the full widthand/or height of the apparatus.

The transmitting surface may also be a receiving surface, arranged toreceive the reflection of the acoustic signal. However, it is envisagedthat, more commonly, the receiver(s) will be separate from thetransmitting surface. Preferably, the apparatus comprises a plurality ofreceivers, each arranged to receive a reflection of the acoustic signalfrom the object directly in front of the display screen, and informationdetermined from all these received signals may be used to characterisethe shape, location or motion of the object.

The use of a transmitting surface in accordance with at least someembodiments of the invention can provide improved accuracy whencharacterising the shape, location or motion of the object, since itreduces the possibility of signal-path confusion which arises whenseparate point transmitters are used. Preferably the transmittingsurface is flat. When this is the case, it will be appreciated that, incontrast to the situation for peripherally-mounted transmitter, anacoustic signal transmitted by the transmitting surface is likely tohave a shorter time of flight (TOF) between the surface, the part of theobject (e.g. a user's fingertip) closest to the surface, and thereceiver, than for any other part of the object. To give an example ofthis, with suitably placed receivers it is possible to mitigate theproblem of the shape, location or motion of a part of the user's handother than an intended fingertip being characterised mistakenly, insteadof that of the intended fingertip.

A further advantage of some embodiments of these aspects of theinvention may be found in a simplification of the calculations necessaryto characterise the shape, location or motion of the object. Inparticular, a simpler, planar trilateration approach, based on thetimes-of-arrival of echoes at each of a plurality of receivers may beused, rather than a spatial intersection-of-ellipses calculation beingrequired, as might be the case for a plurality of separate, pointtransmitters and receivers.

The transmitting surface is preferably sufficiently large that afingertip may be moved in front of it to effect user input to theapparatus, which may be a hand-held portable device. Thus the area ofthe transmitting surface is preferably at least 2 square centimeters;but could be at least 25 square centimeters; and may be 100 squarecentimeters or more. The apparatus may instead be a larger device, suchas a television set; the transmitting surface might therefore beconsiderably larger still; for example, 1 square metre or more.

The transmitting surface could also be used to receive a touch input tothe apparatus; for example, it may additionally make use of knownresistive or capacitative touchpad technology.

In embodiments in which the transmitting surface is disposed insubstantially overlapping relationship with the display screen,preferably at least 50%, more preferably 75%, of the transmitter surfaceoverlaps the display screen. As used herein the proportion of the areaof the transmitter surface considered to overlap the display screen isthat proportion area of the transmitter surface for which a normal tothe transmitter surface passing through the area also intersects thedisplay screen (the latter intersection not necessarily being normal tothe display screen).

The display screen in accordance with the invention could form just acorner of a larger display surface—e.g. with the corner being arrangedfor motion-based interaction with an object and the rest of the displaysurface being passive. However, typically, the display screen of theinvention does not form part of a larger display surface—i.e. all of thedisplay surface acts as a transmitter (of course even in suchembodiments, the apparatus may nonetheless comprise additional, separatedisplay surfaces).

Where the transmitting surface overlaps the display screen it could bedisposed behind the display screen but preferably it is disposed infront of it. This makes it much easier for the sound waves to be emittedunobstructed. Preferably therefore the transmitting surface is fully orpartially transparent, at least in the area thereof which overlaps thedisplay screen. It may, for example, comprise a glass or acrylic sheet;e.g. made of poly(methyl methacrylate). Alternatively, it may comprise athin membrane, for example one made of polyethylene or polyvinylidenefluoride.

The display screen and transmitting surface are preferably both planar,but this is not essential, and one or both may comprise curved surfaces.

The transmitting surface may be in contact with display screen, e.g.over substantially an entire overlap region, or may be substantiallyspaced apart from the display screen, for example, separated by an air-or liquid-gap.

The display screen or larger display surface may use any suitabledisplay technology; for example, LCD, electrophoretic, plasma, ororganic LED.

The object may be anything suitable depending on the application—e.g. anartificial stylus, but more preferably, a human hand or digit. Itsmotion may be characterised by detailed tracking in two or threedimensions (e.g. by determining a sequence of position coordinates).Alternatively, an approach that discriminates between gestures (such asa circling motion of a hand or finger, or a sweep from left to rightacross the display screen), but which does not require such detailedtracking information, may be used.

While the apparatus is arranged to characterise the shape, location ormotion of the object when it is directly in front of the display screen(i.e. on a normal to the display screen), this does not exclude thepossibility of characterisation occurring in other positions. In fact,when a gesture-based interaction mode is being used, it is preferredthat motion of the object is also characterised beyond the regiondefined by a projection of the display screen.

Various mechanisms for transmitting an acoustic signal from thetransmitting surface are possible, some of which are known to theskilled person in other applications, and some of which are believed tobe novel.

In some embodiments, the transmitting surface is arranged to transmit anacoustic signal when an electric potential is applied across it, eitherlengthwise or thickness-wise. To give an example, it may comprisepiezo-electric material.

In some arrangements, it comprises a piezo-electric sheet mounted to arigid backing surface and arranged to vary in thickness on applicationof an electric potential across its thickness. An acoustic signal may betransmitted by controlling the applied potential so as to cause thematerial to vibrate. In this way, an acoustic signal can be transmittedsubstantially uniformly across the whole transmitting surface.

In other arrangements, the transmitted surface comprises apiezo-electric sheet arranged to flex on application of an electricpotential across its length or width.

In some embodiments, the transmitting surface is arranged to transmit anacoustic signal when a varying electric or magnetic field surrounds it.In this case, the apparatus preferably comprises means for generating avarying electric or magnetic field around the transmitting surface.

In some embodiments an actuator is coupled to the transmitting surfaceand is arranged to displace the transmitting surface as a whole. Theactuator may be used to drive the transmitting surface back and forthsubstantially along a normal to the surface so as to transmit theacoustic signal. This is similar to the manner in which a conventionalloud-speaker diaphragm operates. The actuator may comprise a coil and aferromagnetic material arranged to displace the transmitting surfacerelative to the display screen or to a mount. Alternatively, theactuator may comprise one or more piezo-electric members, arranged todisplace the surface. For example, the transmitting surface may besurrounded by a piezo-electric border arranged to move it relative tothe display screen or to a mount.

In some embodiments, the transmitting surface is caused to transmit bythe action of one or more energising transmitters which are arranged todirect acoustic energy through air towards the transmitting surface. Theenergising transmitter or transmitters are preferably located wholly orpartially on the other side of the transmitting surface from the object.The energising transmitters may be configured to induce Lamb waveswithin the transmitting surface, but this is not essential. Such Lambwaves may be primarily extensional or primarily flexural. Lamb wavestravel well within the transmitting surface, and can also couple wellwith the air on the front side of the surface. They are also relativelyresilient to effects such as scratches or damaged parts of the surface.The acoustic signal may be emitted into air substantially uniformlyacross the transmitting surface, or may leave the transmitting surfacemore strongly at one or more points than at others.

An energising transmitter may comprise an acoustic transducer such as apiezo-electric speaker arranged to direct acoustic energy towards thetransmitting surface. The acoustic energy may be focussed on aparticular region of the transmitting surface using, for example, aparabolic reflector. The location and angle of the acoustic energyincident on the transmitting surface may be chosen to induce aparticularly desirable pattern of Lamb waves. In some embodiments, on ormore energising transmitters is controlled so as to cause the acousticsignal to be transmitted directionally from the transmitting surface;for example, by creating a plurality of concentrations of energy withinthe transmitting surface and using beamforming techniques to control thetransmission of the acoustic signal from the transmitting surface.

In alternative embodiments, a plurality of actuators are coupled to thetransmitting surface, each being arranged to generate surface acousticwaves in the transmitting surface. The actuators may be controlled so asto cause the surface acoustic waves to interfere in such a way thatacoustic energy is transmitted directionally from the transmittingsurface and/or is transmitted primarily from a particular point on thetransmitting surface. This can be achieved when the movement in thesurface induced by the interfering surface acoustic waves is sufficientfor coupling of energy into the air. Coupling between surface acousticwaves and air can often be in the range of 1% of the energy travellingalong the surface, depending on the material of the surface. Thus,energy needs to be focussed well in order to provide sufficient waveamplitudes for, say, tracking an object moving in the air over or in thevicinity of the surface. The location of a point from which signals aretransmitted into the air may be variable, and control means may beprovided which is configured to transmit sound more strongly from apoint relatively close to the object than from another point on thetransmitting surface which is relatively removed from the object. Thecontrol means may alternatively or additionally be configured to directa beam of sound towards the object using beamforming techniques.

The acoustic signal may comprise one or more chirps, but in somepreferred embodiments it comprises one or more pulses. The inventor hasdiscovered that, at least in some arrangements, pulses can be moreefficiently transmitted by the transmitting surface than chirps.Moreover, the non-linear amplifiers needed to generate pulses or pulsetrains are typically easier to develop and fabricate than amplifierssupporting continuous wave-forms.

Any of the methods described with reference to earlier aspects mayequally be used with these aspects also. In particular, when theapparatus comprises a plurality of transducers, a step of determining asubset of the transmitter-receiver combinations which give rise to areceived signal meeting a predetermined clarity criterion may beperformed. Beamforming using the transmitting surface alone or incombination with other transmitters, may also be performed.

In all of the methods and devices described herein, results, such as ashape estimate (e.g. contour coordinates) or a position estimate of anobject, may be stored in a volatile or non-volatile memory. Additionallyor alternatively they may be displayed on a display device. A displaysignal for a display device may be provided. Additionally oralternatively methods described herein are used to control a device.They thus could comprise the step of providing a control signal for adevice.

The methods of the invention are preferably carried out using computingmeans, computing machines, data processing apparatus or any other devicecapable of carrying out stored instructions. Such a device may bestatic; although the invention can equally be used with mobile devices.Although certain aspects have been described with reference to stepscarried by a device, it will be appreciated that a part or all of one ormore of these steps may, in some less-preferred embodiments, be carriedout on a remote processor, such a network server, which is incommunication with the device (e.g. over an RF channel); sucharrangements are within the scope of the present invention.

Any of the definitions and features, optional or otherwise, describedherein in the context of one aspect of the invention may, whereappropriate, be applied to any of the other aspects also.

Certain preferred embodiments of the invention will now be described, byway of example only, with reference to the accompanying drawings, inwhich:

FIG. 1 is a perspective drawing of a user interacting with a device inaccordance with the invention;

FIG. 2a is perspective drawing of a part of the device;

FIG. 2b is a graph of the signal received by a receiver of the device;

FIG. 3 is a diagram of two impulse responses calculated from receivedsignals;

FIG. 4 is a figurative, partial side view of a second device accordingto the invention, illustrating a principle of its operation;

FIG. 5 is a more-detailed figurative, partial side view of the seconddevice;

FIG. 6 is a figurative, partial side view of a third device according tothe invention;

FIG. 7 is a figurative, partial side view of the second device operatingin a first mode;

FIG. 8 is a figurative, partial side view of the second device operatingin a second mode.

FIG. 9 is a figurative, partial side view of a fourth device accordingto the invention; and

FIG. 10 shows in more detail how the position of the fingertip can beestimated above a display screen.

FIG. 1 shows a static device 2 comprising a display screen 4 and asurrounding frame 6. Flush-mounted on the frame are a top array 8 oftransducers and a left-side array 10 of transducers. The top array 8comprises two parallel rows 12, 14 of transducers. The transducers alongeach row are alternately a transmitter followed by a receiver. Thetransmitters in the upper row 12 are aligned vertically with those ofthe lower row 14. The left-side array 10 comprises three parallelcolumns 16, 18, 20 of transducers. Again, transmitters and receiversalternate down the columns, with transducers of the same type beinghorizontally-aligned between the columns. The top-left corner regioncontinues the pattern of the left-side array 10, but it couldalternatively be a continuation of the top array 8. Of course, otherpatterns of transmitters and receivers are possible; however havingtransducers of the same type aligned perpendicularly to the axis of thearray is preferred as it enables accurate steered propagation- or‘beamforming’ about the axis of the array (i.e. beams can be directedbeyond the edges of the frame 6). There may also be similar arrays (notshown) on the right-side of the screen 4 and at the bottom of thescreen.

A user's right hand 22 is near the screen 4, interacting with thedevice, for example to direct a cursor (not shown) around the screen.However the user's left hand 24 (or equally the hand of a second user)is encroaching from the left side, moving in the direction of the arrow.For at least some of the transducer combinations in the arrays 8, 10,the right hand 22 and left hand 24 are at overlapping time-of-flightdistances. This means that the motion path of the right hand 22 cannotreadily be separated from that of the left hand 24, at least for thosetransducer combinations.

The location or movement or shape of the hand is not therefore readilydiscernable using conventional techniques.

Accordingly, the device transmits a signal from one of the transmittersand listens for responses at some or all of the receivers. Thosereceivers that exhibit a clean signals; i.e. one with a clear leadingedge (e.g. from the extended finger tip of the hand 22), possiblyfollowed by a later, second edge (e.g. from another part of the hand 22such as a knuckle), are noted in some way (e.g. in a memory array on thedevice). Those that are ‘smeared’ due to overlapping signals from otherobjects, such as from the left hand 24, are not noted in the same way.

Once this is done, a signal is transmitted from another of thetransmitters, and the received signals are analysed as before. This maybe repeated for every transmitter, until a matrix oftransmitter-receiver combinations that are ‘clean’ has been formed (thismatrix can be an abstract construct which may be manifested physicallyusing any suitable data structure in a volatile or non-volatile memoryof the device 2, or in any other appropriate way). The matrix entriesmay be binary (whether a clarity condition has been met or not) or mayquantify a degree of clarity. It may not be necessary or appropriate toinvolve all the transmitters and all the receivers in this process: insome circumstances a subset of each may be used.

A image or map of the right hand 22 is then constructed usingbeamforming imaging techniques, but only using thosetransmitter-receiver combinations that were noted as ‘clean’ in thepreceding stage. Of course, the image may be represented in the device'smemory in any appropriate way, and is not necessarily stored in arecognised image file format.

Any suitable beamforming technique may be used, but the presentembodiment employs delay-and-sum beamforming. Conceptually, the spacearound the device is divided into voxels (volumetric pixels), thecentre-points of which are represented as 30 coordinate vectors. Each ofthese centre-points is at a specific time-of-flight distance for eachtransmitter-receiver combination. For each transmitter-receivercombination selected in the preceding step, a signal is transmitted fromthe transmitter and the response at the receiver is analysed (onetransmit signal may be used by several transmitter-receiver combinationsto allow a high update rate). An impulse response is calculated for thatcombination. For each voxel centre-point, the contributions of therelevant part of the impulse responses of each of the selectedtransmitter-receiver combinations (i.e. the response at or around theappropriate time-of-flight distance for each combination) are summedtogether. Preferably, rather than summing the calculated impulseresponses, which can result in positive and negative values cancellingout, the envelope or local energy of the impulse response is used in thesumming step.

The device now has some information relating to the shape of the righthand 22, determined from the presence or absence of reflections at eachvoxel, as well as from information relating to the intensity of thereflection at each voxel, and possibly also information relating toDoppler shift or other factors. This information can then allow thedevice to perform transmit beamforming with some or all of thetransmitters of the arrays 8, 10. This can be accomplished by the devicefirst deriving an estimated outline of the hand 22 and then coordinatingthe transmission of sound from a plurality of transmitters so as todirect a beam towards all or a part of the hand 22. In one mode, thedevice transmits a relatively narrow beam towards a part of the hand 22;for example, an extended fingertip. This allows information about theshape and/or position of that part of the hand 22 to be determined to ahigher degree of accuracy (for example, using smaller-scale voxels) thanin the initial stages. The beam may be directed at different parts ofthe hand 22 at different times; it may, for example, be sweptprogressively across the whole of the object in a scanning pattern, orit may be directed towards parts of particular relevance to a userinput. For example, if a pinching movement is identified as occurring,transmit beams may be directed alternately towards the tip of the thumband towards the tip of the index finger, in order to characterise themotion of these parts in more detail.

Similarly to before, the device may, for each transmit beam, determinewhich receivers provide a ‘clear’ signal. Effectively a matrix may beformed with transmitter groups on one axis and receivers on the otheraxis (of course, not all possible groups of transmitters and not allreceivers need be included in this matrix), recording whether thatcombination results in a signal satisfying a clarity condition. Thisdetermination may be similar to that set out above with respect to thematrix of individual transmitter-receiver pairs, possibly modified toallow for the different situation of having a plurality of transmitters;however, it is preferably still related to whether a clear leading edgeis detectable, or two leading edges in succession. Rather than simplynoting a binary result (whether the clarity condition is met or not),the matrix entries may record a degree of clarity on a quantitative orqualitative scale.

Once a set of receivers that give clear results with certain transmittergroups has been determined, these combinations are used to construct orrefine a voxel-based 3D image as before. The image may be of higherspatial resolution since, by beamforming the transmit signal, a bettersignal-to-noise ratio is typically achieved. It is not necessary tocompute a full 3D voxel image; in some preferred embodiments, a sparsevoxel representation, or list of voxels which are non-zero, is used.This can save memory and thus reduce overall system costs.

In a similar manner, some or all of the receivers may be groupedtogether to perform receive-side beamforming of the received signals.This may be performed in addition to transmit beamforming, in which casethe receive focus will normally by directed at the same region as thetransmit beam. Again, a matrix of transmitters against receiver-groups,and/or of transmitter-groups against receiver-groups, may be constructedand used to determine which combinations to use.

As the hand 22 moves, or as other objects appear or disappear, theclarity of various transmitter-receiver combinations (whether singly orin beamforming groups) is likely to change, as it is bound up in thephysical reality beyond the device 2 (e.g. whether to objects are atsimilar time-of-flight distances as each other for certaincombinations). The channel-clarity determining steps are thereforepreferably repeated intermittently (for example, according to aschedule, or when noise or errors in the results are determined toexceed a threshold). The clarity determination has here been describedas a separate temporal phase; however it is possible that the same datathat is used to construct a voxel-based image is also used to determinethe clarity of the combinations; for example, while some receivers arebeing used to determine information relating to the shape or position ofthe hand 22 (being those that are clear), signals from the otherreceivers may nonetheless still be being analysed in order to determinewhether any of those becomes clearer than any currently in active use.Unlike transmit beamforming, where a plurality of simultaneous beams hasthe potential to introduce undesired noise, the received signals may bebeamformed in any direction without causing any interference on anyother beamforming operation. Where processing power is sufficient, aplurality of receive-side beams may be used simultaneously from the sameor different receivers.

The location and shape of the hand 22 can be used to control a functionof the device 2; for example, to control a music player function of thedevice (e.g. raising and lowering volume as a fingertip of the hand 22is moved up and down the screen 4, respectively).

FIGS. 2a and 2b exemplify the use of receive-side beamforming tosuppress energy from some angles while retaining energy from others.FIG. 2a shows a portion of the left-side array 10. The line 202indicates an axis of the array. The angles from −90 degrees to +90degrees in the horizontal axis of FIG. 2b are defined with reference tothis axis 202. The vertical axis of FIG. 2b represents the intensity ofthe processed signal that results from beamforming using the receiversin the left-side array 10. The signals from the direction of the righthand 22 (i.e. to the right of a vertical plane through axis 202 whenlooking at FIG. 1) are received at high intensity, while those from thedirection of the intruding left hand 24 are suppressed. The actualcalculation of the beamforming parameters (e.g. the delays to apply toeach receiver's signals) is performed in any suitable way; for example,by first detecting the presence of the left hand 24 and then activelyplacing zeros or near-zeros or energy cancellation in those directions,while actively placing steering vectors in the direction of the righthand 22. Energy from certain directions can be cancelled out bydesigning the beamforming scheme as a spatial filter, letting signalsfrom some directions come through while blocking others. Adaptive, i.e.data-driven cancellation, such as the minimum-variance method, or Caponor Apes or specific null-steering methods can be employed. The lattercan be constructed by creating a filter, receiving input from all theinput channels, and suitably delaying, weighting and summing the inputsso that the desired directivity for steering and cancellation isobtained. The filter can be implemented in the time-domain,frequency-domain, time-frequency domain, Fractional Fourier domain,chirp domain or any other suitable domain. Adaptive beamforming may thusbe used to steer in the direction of the right hand 22 while suppressingall other directions. Traditionally, adaptive beamforming schemes arecomplex and costly However the present arrangements are generallysimpler and can therefore provide substantial reductions in CPU costs,making adaptive beamforming schemes feasible in low-cost, hand-helddevices.

FIG. 3 shows what the received impulse response 301 from a channel orcombination meeting a clarity criterion looks like, in comparison to theimpulse response 303 from one that does not. The impulse response 301has a section 302 that exhibits a short, clear pulse, followed by aperiod 304 of relatively low energy, before a section 305 where theenergy level starts fluctuating. This clear signal portion 302 followedby low energy followed by further signals (which may or may not have aclear leading edge) is identified by the device 2 as indicative of acombination that contains information; i.e. reflections that are likelyto come from a single object or part. By contrast, the impulse response303 shows no clear isolated signal part and no clear leading edge, andthe corresponding combination is likely not to be used in the subsequentimaging.

Multiple successive impulse responses are preferably analysed togetherby composing them into an impulse response “image”, in whichconsecutive, discretely-sampled impulse responses are alignedside-by-side to form a two-dimensional array (or image if the arrayvalues are represented as greyscale pixels). The detection of a peak orleading edge or otherwise interesting part in an impulse response imagecould happen in any number of ways.

For instance, a leading edge may be detected using a leading edgedetector which moves a sliding frame around the impulse response image,computing the ratio of the maximum amplitude and the median or averageamplitude within the frame.

Another approach to detecting a leading edge is to move a sliding windowdown an impulse response 301, where the window is divided into an upperwindow and a lower window. If the energy levels in the upper window issignificantly less than in the lower window, an edge is detected. Thetest as to whether one set of amplitudes is higher than another (i.e.whether the energies in the upper window are greater than those in thelower) can be conducted using a statistical test to check if the mean ofone population is significantly above the mean of a second population;for example, by assuming normal distributions and using at-test. Apossible better way to detect a leading edge is to use a constant falsealarm rate (CFAR) filter, as described in “Statistical SignalProcessing” by L. L. Scharf, chapters 4.9-4.12. The CFAR filter can beused to examine the presence of a known signal in unknown levels ofnoise. The known signal, or even a linear space of known signals, wouldhere be a set of empirically-observed leading-edge signals, such animpulse response 302 known to contain a clear reflection, plus othersimilar and possibly phase-shifted signals. These provide alow-dimensional space of signals known to exhibit desirable leading-edgecharacteristics.

The CFAR subspace filter then provides a statistical test to whether agiven impulse response 302, 303 contains a leading edge or not. Thistechnique is particularly beneficial when working with impulse responsesignals, since the vast dynamic range may require a large number of bitsto represent, with both the least-significant bits (LSB) and themost-significant bits (MSB) carrying information. The total number ofbits can be magnitudes larger than the bit-rate of the sampling system.Hence, precise, but sometimes highly fluctuating values, representbackground noise and foreground objects, which makes it difficult toprovide exact thresholds defining the foreground and background, as iscustomary in other fields, such as image processing.

The CFAR-filter, however, is invariant to scaling of the match signal;rather, it obtains a uniformly most powerful test limit for a givenfalse-alarm probability, and its probability of misses is dependent onlyon the signal-to-noise ratio. Moreover the CFAR filter can be extendedto work for multiple impulse responses (i.e. multiple time frames for asingle channel), and using a 20 match filter mask, such as aline-filter, can provide even more robust detection of a leading edge.The CFAR filter can further be extended to work over multiple time-stepsand multiple channels, i.e. a 3D CFAR filter, or over multiple channelsover a single time-frame, i.e. another representation of a 2D CFARfilter.

The use of a CFAR-filter with impulse response images also enablesbetter detection of channels which are “clear”. Once the clear channelshave been detected, the system can decide to use only certain parts(e.g. a certain time frame after signal transmission) of the impulseresponse signal for imaging. Typically, the parts of the impulseresponse which are not informative, i.e. not “clear”, are kept out ofthe imaging computations.

More generally, it is possible to inspect an impulse response for partshaving a high level of information or entropy. A leading edge is onetype of information. Generally however, a sliding window inspection ofan impulse response can be used to decide which subparts of the responseare “informative”, i.e. by studying the distribution of taps. Suitablemeasures include, among others: negentropy; Kullback-Leibler divergencesin temporal, spatial or other domains; degree of match with contrastfunctions such as skewness or kurtosis; and measures of sub- orsuper-Gaussian distributions.

A “clear” channel can also be detected in terms of its self-consistency.To detect such self-consistency, a similar approach can be adopted tothe previously-described algorithm for imaging using voxels; but it canbe preferable in this context to use multiple transmissions rather thana single transmission, so that impulse responses can be studied inunison. Self-consistency may be determined by autocorrelation or anyother suitable method. The channels and the channel time frames showinga high degree of self-consistency can then be used for imaging.

FIGS. 4 to 9 show various embodiments having a transmitting surface,which either comprises or overlaps a display screen. Such an arrangementfacilitates positioning of the leading fingertip, since the fingertipbecomes the ‘leading point’ relative to the transmit source.

FIG. 4 shows an embodiment in which an acoustic signal (indicated byfour arrows) is transmitted outwardly from the surface of a displayscreen 430 towards a user's hand 22. In this illustration it can be seenthat, as will normally be the case, the user's fingertip 22A is theclosest point of the user's hand 22 to the screen 430. Two microphones432, 434 are located on either side of the display screen 430, andreceive respective echoes from the user's hand 22, and fingertip 22A inparticular. By determining the times-of-flight (TOF) of the acousticsignal from the display screen 430 to the fingertip 22A and to therespective microphones 432, 434, information relating to the position ofthe fingertip 22A can be determined.

Although the user's thumb 22B may be at a shorter TOF distance from thedisplay screen 430 and the right-side microphone 434 than the user'sfingertip 22A, nonetheless when the timings of both microphones 432, 434are combined, a processor in the apparatus (not shown) can nonethelessdetermine that the fingertip 22A is closer to the screen surface thanthe thumb 22B are therefore track the fingertip 22A for a userinteraction, such as controlling the position of a cursor shown on thedisplay screen. This ability to identify a point on the user's handnearest to the screen surface is facilitated by the acoustic signalbeing transmitted from across the entire area of the display screen,rather than from a point transmitter. In particular, when the displayscreen 430 is bordered by several microphones, it is more likely that asignificant number of them will satisfy a clarity condition than if theacoustic signal were emitted from a single point.

FIG. 5 shows the mechanism by which the display screen 430 is caused totransmit the acoustic signal. An acoustic exciting transducer 436 issituated behind and spaced away from the screen 430, pointed towards theback of the screen at an angle a. The exciting transducer 436 isarranged to transmit a pulse or a continuous signal into the displayscreen 430. The transducer 436 may be a conventional electrostaticultrasound transducer, such as the Series 600 or Series 7000 transducersfrom SensComp, Inc. Preferably, however, it is a cMUT (capacitativemicromachined ultrasonic transducer). One such transducer is describedin “The design and characterization of micromachined air-coupledcapacitance transducers” by D. W. Schindel et al., IEEE Trans.Ultrasonics and Ferroelec. Freq Control 42 (1995), pp. 42-50.

FIG. 6 shows a variation of the embodiment of FIG. 5 in which, insteadof the exciting transducer 636 being directed towards the display screen630, it is directed towards a layer of a transparent medium 638 which isplaced on top of, and spaced away from, the display screen 630. Thetransparent medium 638 may be made of a membrane or plate, for exampleof acrylic glass; i.e. poly(methyl methacrylate). In the embodiment ofFIG. 6, the exciting transducer 636 is hidden from view behind asurrounding frame 640 which borders the display screen 630. This leadsto a more elegant appearance and allows a reduction in the overallthickness of the screen assembly. A similar frame arrangement could beused in other embodiments also.

Acoustic receivers are not depicted in FIG. 6, but could be locatedbehind grilles in the frame 640. The transparent medium 638 can beattached to the screen 630 or the frame 640 by any suitable means; forexample, a thin bolt, or two or more miniature pillars, support bars, orsupporting springs, thereby creating a set of free-free boundaryconditions. The transparent medium 638 may alternatively extend into asupport groove or slot (not shown) of greater width than the thicknessof the medium, such that the medium has freedom to move within theconfines of the groove. The groove may surround the entire perimeter ofthe medium 638, or just a part of it. The transparent medium is thusretained between the exciting transducer 636 and the frame 640, yet hasfreedom to vibrate. Alternative support arrangements may be used.Various suitable configurations are described in “Formulas for NaturalFrequency and Mode Shape” by Blevins, R. D. (1979, Malabar, Fla.:Krieger).

FIGS. 7 and 8 illustrate the waves that are excited in the screen by theexciting transducer 436. The exciting transducer 436 transmits acousticenergy towards the rear face of the display screen 430. This causes awave to propagate in the display screen 430 (or, in alternativeembodiments, in a transparent medium overlying the display screen).These waves are indicated by the waveforms 440 and 442 (not to scale).The waves couple into the air in front of the display screen 430,thereby causing an acoustic signal to be transmitted towards the user'shand 22.

The waves arising in the display screen 430 are Lamb waves (guidedacoustic waves in plates). They are solutions to the wave equation forlinear elastic waves, subject to boundary conditions defining thegeometric structure of the display screen 430. The waves are highlydispersive, meaning that the wave speed depends on the frequency. Thisstands in contrast to acoustic waves propagating in air. The solutionsto the Lamb wave equations represent the kinds of wave that canpropagate, based on the properties of the medium and the boundaryconditions. They belong to two distinct families: symmetric orextensional-mode waves, in which the upper-surface waveforms 440 mirrorthe lower-surface waveforms 442, as shown in FIG. 7; and anti-symmetricor flexural-mode waves, in which the upper-surface waveforms 440 are inantiphase with the lower-surface waveforms 442, as shown in FIG. 8.

To generate these Lamb waves, the angle α of the exciting transducer 436to the display screen 430 must match the critical angle. Critical anglesare explained in detail in the paper “High contrast air-coupled acousticimaging with zero group velocity lamb modes”, by S. Holland and D. E.Chimenti, Elsevier Ultrasonics, Vol. 42, 2004, pp. 957-960. For mostfrequencies, there are only a few, discrete, incident angles satisfyingthe phase match criterion; i.e. which are able to excite Lamb waves.However, at the zero-group-velocity frequency, there is a wider range ofangles α for which the energy from the exciting transducer 436 coupleseffectively at the same frequency. Therefore, if a focussed excitingbeam, spanning a range of angles, at the zero-group-velocity frequencyis incident on a plate (such as the display screen 430), the entirerange of angles near the zero-group-velocity point is transmittedefficiently from the air to the plate, and also through the plate to theair on the opposite side, at that frequency. This leads to adramatically higher transmission into the air at the front of thedisplay screen 430 than is the case for other transmission modes.Preferably therefore, the frequency and the angle of incidence a arechosen so as to match the zero-group-velocity frequency, and theexciting transducer 436 has an angle of inclination to match with thezero-group-velocity mode.

Several exciting transducers may be arranged behind the display screen430, for example around its periphery, all transmitting into the displayscreen 430 (or, equivalently, into a transparent material, such asacrylic glass, overlaying the display screen). By exciting several suchtransducers, or selectively employing a subset of the transducers,directive transmission into the display screen 430 can be accomplished,thereby forming points or areas of particularly high intensity in thedisplay screen 430. This causes the acoustic signal to be transmittedinto the air in front of the display screen 430 not uniformly across thewhole surface of the display screen, but with particular intensity in aselected region of the screen 430. This can be employed to scanspatially the area in front of the display screen, creating strongerreflections or “virtual transmission points” from certain zones in frontof the screen 430 than from others.

FIG. 9 shows an alternative embodiment which employs a different way ofusing the screen as an ultrasonic transmitter. A foil 944 is situatedadjacent the front face of the display screen 930, with an air gapbetween the foil 944 and the screen 930. It can be made to vibrateacoustically in a piston-like fashion using a piezo-actuator, i.e.vibrating the entire foil 944 as a whole. The foil 944 has a thicknessof between 0.1 to 2.0 mm and sits 1 to 10 mm away from the front of thedisplaying surface. In some embodiments, in addition to being able totransmit an ultrasonic acoustic signal, the foil 944 may be used totransmit audible sound for audio applications; for example, it may beused for playing the soundtrack to a movie being shown on the display.Suitable foils or membranes, and approaches to arranging them in frontof a display screen, are described in US 2009-0285431, U.S. Pat. No.6,720,708 and U.S. Pat. No. 7,038,356. These disclosures aim to produceaudible sound; however the present inventor has realised that suchapproaches may conveniently be adapted for use in the present invention.

FIG. 10 shows in more detail how the position of the fingertip 22A canbe estimated above a display screen 1030 using any one of the abovephysical arrangements for transmitting a signal from the display screen1030.

An acoustic signal (e.g. a pulse or chirp) is emitted from the screen1030. This is reflected off the fingertip 22A, and is received by thereceiver 1034. The time-of-flight of the sound moving from the surfaceof the display screen 1030 to the fingertip 22A and on to the receiver1034 is measured. This quantity can be estimated by detecting aleading-edge of the echo in the received signal. Alternatively, it couldbe computed from a calculated impulse response signal; i.e. not directlyfrom the raw received signal. The emitted signal could be a pulse, achirp, a continuously or continually transmitted signal, a pulse train,or any other suitable signal; and an impulse response may be calculatedtherefrom.

The position of the fingertip 22A cannot be unambiguously resolved byusing a single receiver alone. Nonetheless, with a single receiver 1034,multiple time-of-flight estimates can be used to infer the position ofthe finger in 3-space, by using a Pythagoras-like-principle. Consideringthe single receiver 1034 shown in FIG. 10, the height of the fingertip22A from the screen is denoted by y; the distance from the receiver 1034to the point on the display screen 1030 nearest the fingertip 22A, by x;and the length of the direct path between the fingertip 22A and thereceiver 1034, by w.

Suppose a time-of-flight value is measured for the receiver 1034, equalto a distance k.Then k=w+y, and so w=k−y. By Pythagoras' theorem,

w=√{square root over (x² +y ²)}

and hence

(k−y)2=x ² +y ².

Rearranging, this gives:

k² − 2k.y ⋅ +y² = x² + y² k² − 2 ky = x² − 2 ky = X² − k²$y = {\frac{k}{2} - \frac{x^{2}}{2\; k}}$

Thus the point (x, y) lies on a parabola determined by the measuredvalue k. If there were more channels available, the point (x, y) couldbe worked out as intersection point, or, when more channels than theminimum are available, by an approximation that could be computed i.e.by iterative means using a steepest-descent, gradient search, conjugategradient, simplex or other method for solving the approximation problem.This two-dimensional example assumes that the fingertip 22A is known tobe in or adjacent a plane perpendicular to the display screen 1030, sothat the position of the fingertip 22A along a z-axis, perpendicular tox and y, is unimportant.

In the more common situation of three-dimensional sensing, where adetermination of the coordinates (x, y, z) is desired, the coordinate xcan be replaced by the term √{square root over (x²+z²)} in the equationsabove, and it will be seen that the corresponding intersection surfacesare revolutions of the parabola functions around an axis through thereceiver element 1034, but limited by the edges of the screen. Theposition of the fingertip is then derived by considering theintersection of three or more such surfaces in 3-space. If receivers ofshapes other than an effective point receiver are used—for example, ifthe display screen 1030 is also a receiving surface, or if an elongatereceiving element were used—then a different set of geometric equationswould arise. In a simplified embodiment, using the same surface as atransmitter and a receiver could be used to detect a situation where theuser is lifting his finger from the surface or pushing it down on it.

Thus methods have been described herein for detecting and using a subsetof channels in order to generating useful estimates from a scene.Arrangements in which an acoustic signal is transmitted from atransmitting surface have also been described.

1. An apparatus for characterising the shape, location or motion of anobject, said apparatus comprising: a display screen; a transmittingsurface comprising the display screen, or being disposed insubstantially overlapping relationship with the display screen, whereinthe transmitting surface is arranged to transmit an acoustic signal fromsubstantially all of the transmitting surface, the acoustic signaltravelling through air towards an object located directly in front ofthe display screen; a receiver arranged to receive a reflection of theacoustic signal from the object; and a processing system configured tocharacterise the shape, location or motion of the object usinginformation determined from the received reflection of the acousticsignal.
 2. The apparatus of claim 1, wherein the transmitting surface isalso a receiving surface, and the receiver is arranged to receive thereflection of the signal at the receiving surface.
 3. The apparatus ofclaim 1, further comprising: a plurality of receivers separate from thetransmitting surface, each receiver of the plurality of receivers beingarranged to receive a reflection of the acoustic signal from the object.4. The apparatus of claim 1, wherein all of the display screen acts as atransmitter.
 5. The apparatus of claim 1, wherein at least 75% of thetransmitter surface overlaps the display screen.
 6. The apparatus ofclaim 1, wherein the display screen is an LCD screen, or anelectrophoretic display screen, or a plasma display screen, or anorganic LED display screen.
 7. The apparatus of claim 1, wherein thetransmitting surface is disposed in front of the display screen.
 8. Theapparatus of claim 1, wherein at least an area of the transmittingsurface is optically transparent.
 9. The apparatus of claim 1, whereinthe display screen is planar.
 10. The apparatus of claim 1, wherein thetransmitting surface is planar.
 11. The apparatus of claim 1, whereinthe transmitting surface is arranged to transmit an acoustic signal whenan electric potential is applied across the transmitting surface. 12.The apparatus of claim 1, wherein the transmitting surface is arrangedto transmit an acoustic signal when a varying electric or magnetic fieldsurrounds the surface.
 13. The apparatus of claim 1, further comprising:an actuator coupled to the transmitting surface and arranged to displacethe transmitting surface as a whole.
 14. The apparatus of claim 1,further comprising: an energising transmitter, arranged to directacoustic energy through air towards the transmitting surface.
 15. Theapparatus of claim 14, wherein the energising transmitter is located atleast partially on the opposite side of the transmitting surface to theobject.
 16. The apparatus of claim 14, wherein the energisingtransmitter is configured to induce Lamb waves within the transmittingsurface.
 17. The apparatus of claim 14, further comprising: means tofocus acoustic energy from the energising transmitter on a particularregion of the transmitting surface.
 18. The apparatus of claim 1,further comprising: a plurality of actuators coupled to the transmittingsurface, each actuator being arranged to generate surface acoustic wavesin the transmitting surface, and wherein the apparatus being configuredto control the actuators so as to cause the surface acoustic waves tointerfere in such a way that acoustic energy is transmitteddirectionally from the transmitting surface.
 19. The apparatus of claim18, further being configured to control the actuator so as to cause thesurface acoustic waves to interfere in such a way that acoustic energyis transmitted primarily from a particular point on the transmittingsurface.
 20. The apparatus of claim 1, further comprising: a controllerconfigured to direct a beam of sound towards the object.
 21. Anapparatus for characterising the shape, location or motion of an object,said apparatus comprising: a display screen arranged to transmit anacoustic signal, through air, towards an object located in front of thedisplay screen; a receiver arranged to receive a reflection of theacoustic signal from the object; and a processing system configured tocharacterise the shape, location or motion of the object usinginformation determined from the received reflection of the acousticsignal.
 22. A method of characterising the shape, location or motion ofan object, said method comprising: transmitting an acoustic signal fromsubstantially all of a transmitting surface, the transmitting surfacecomprising a display screen or being disposed in substantiallyoverlapping relationship with a display screen, the acoustic signaltravelling through air towards an object located directly in front ofthe display screen; receiving a reflection of the signal from theobject; and characterising the shape, location or motion of the objectusing information determined from the received signal.
 23. The method ofclaim 22, wherein the object is all or part of a human hand.